

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=iso8859_1" />
    
    <title>Tutorial</title>
    <link rel="stylesheet" href="../../_static/default.css" type="text/css" />
    <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
    <link rel="top" title="SOM Analyst v1 documentation" href="../../index.html" />
    <link rel="prev" title="Send Email" href="uiEmailMe.html" /> 
  </head>
  <body>
    <div class="related">
      <h3>Navigation</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="../../genindex.html" title="General Index"
             accesskey="I">index</a></li>
        <li class="right" >
          <a href="../../py-modindex.html" title="Python Module Index"
             >modules</a> |</li>
        <li class="right" >
          <a href="uiEmailMe.html" title="Send Email"
             accesskey="P">previous</a> |</li>
        <li><a href="../../index.html">SOM Analyst v1 documentation</a> &raquo;</li> 
      </ul>
    </div>  

    <div class="document">
      <div class="documentwrapper">
          <div class="body">
            
  <div class="section" id="tutorial">
<h1>Tutorial</h1>
<p>This tutorial contains step-by-step instructions on how to use the provided example dataset with SOM Analyst in ArcMap. The source data set for this tutorial is provided with SOM Analyst and is located in its sub-folder named <strong>dat</strong>. The file named <strong>census.csv</strong> contains gender, age, race, and housing data for each U.S. population census between the years 1900 and 1990.</p>
<p>First, the data is converted from the comma separated file format (.csv) to the database file format (.dbf) so that normalizations can be performed. Second, the raw count data are normalized by state population counts. Third, every variable is normalized into a 0 to 1 range and the preprocessed data are then exported to the SOM input format. Using those input data, a SOM is trained in two stages. The input data are then projected onto the finished SOM. Finally, a number of visualizations are produced.</p>
<div class="section" id="system-requirements">
<h2>System Requirements</h2>
<ol class="arabic simple">
<li>Windows (any version)</li>
<li>ArcGIS 9.3 (legacy toolboxes for ArcGIS 9.0-9.2 are provided, but untested)</li>
<li>Python 2.5 (included in the default ArcGIS 9.3 installation)</li>
</ol>
</div>
<div class="section" id="download">
<h2>Download</h2>
<p>SOM Analyst is available for download from <a class="reference external" href="http://somanalyst.googlecode.com">http://somanalyst.googlecode.com</a></p>
</div>
<div class="section" id="adding-the-toolbox">
<h2>Adding the Toolbox</h2>
<p>Add the SOM Analyst Toolbox to ArcGIS.</p>
<ol class="arabic simple">
<li>Open the ArcToolbox panel by clicking on the <strong>Window</strong> menu and select <strong>ArcToolbox</strong>. Alternatively, click on the toolbox icon on the menu bar.</li>
</ol>
<div class="figure">
<img alt="../../_images/ArcToolbox.png" src="../../_images/ArcToolbox.png" />
</div>
<ol class="arabic simple" start="2">
<li>Right click in the ArcToolbox panel and select <strong>Add Toolbox...</strong>.</li>
</ol>
<div class="figure">
<img alt="../../_images/AddToolbox.png" src="../../_images/AddToolbox.png" />
</div>
<ol class="arabic simple" start="3">
<li>Browse to the location of SOM Analyst and select <strong>guiArcGIS93.tbx</strong> and click <strong>Open</strong>.</li>
</ol>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">Depending on your computer setup, it may be necessary to first &#8220;connect&#8221; to the folder that contains SOM Analyst. In that case, click in the dialog box on the icon of a folder with an arrow pointing to a globe.</p>
</div>
<div class="figure">
<img alt="../../_images/guiArcGIS93.png" src="../../_images/guiArcGIS93.png" />
</div>
<p>The SOM Analyst toolbox is now accessible through the ArcToolbox panel.</p>
<div class="figure">
<img alt="../../_images/SOManalyst.png" src="../../_images/SOManalyst.png" />
</div>
<p>Browse through the toolbox to familiarize yourself with the tools.</p>
<div class="figure">
<img alt="../../_images/ToolList.png" src="../../_images/ToolList.png" />
</div>
</div>
<div class="section" id="convert-data-format">
<h2>Convert Data Format</h2>
<p>Convert the data to a database file format.</p>
<ol class="arabic simple">
<li>Run the <strong>Data File to Database File</strong> tool by double clicking on it in the <strong>File Format Conversions</strong> toolbox of the <strong>Data Preprocessing</strong> toolbox.</li>
</ol>
<div class="figure">
<img alt="../../_images/toXbase.png" src="../../_images/toXbase.png" />
</div>
<ol class="arabic simple" start="2">
<li>Select <strong>census.csv</strong> as the <em>input data file</em>.</li>
<li>Set <strong>Comma Separated Values (CSV)</strong> as the <em>input file format</em>.</li>
<li>Change the <em>output database file</em> to <strong>census.dbf</strong>.</li>
<li>Click <strong>OK</strong> to run the conversion.</li>
</ol>
<p>In the table properties the data type for each column is text.</p>
<div class="figure">
<img alt="../../_images/censusfields.png" src="../../_images/censusfields.png" />
</div>
<p>The values in the table are left justified indicating that they are text.</p>
<div class="figure">
<img alt="../../_images/census.png" src="../../_images/census.png" />
</div>
</div>
<div class="section" id="normalize-data">
<h2>Normalize Data</h2>
<p>Normalize values in the database file.</p>
<ol class="arabic simple">
<li>Run the <strong>Normalize by Variable</strong> tool by double clicking on it in the <strong>Value Transformations</strong> toolbox of the <strong>Data Preprocessing</strong> toolbox.</li>
</ol>
<div class="figure">
<img alt="../../_images/normalize.png" src="../../_images/normalize.png" />
</div>
<ol class="arabic simple" start="2">
<li>Select <strong>census.dbf</strong> as the <em>input database file</em>.</li>
<li>Select <strong>Population</strong> as the <em>normalize by column</em>.</li>
<li>Change the <em>output database file</em> to <strong>census.dbf</strong>.</li>
<li>Select the columns <strong>male</strong>, <strong>female</strong>, <strong>Under_15</strong>, <strong>15_64</strong>, <strong>65_Over</strong>, <strong>Am_Indian</strong>, <strong>Asian</strong>, <strong>Black</strong>, and <strong>White</strong> in the <em>columns to normalize</em> field.</li>
<li>Click <strong>OK</strong> to run the normalization.</li>
</ol>
<p>The resulting table contains population ratios.</p>
<div class="figure">
<img alt="../../_images/normalizevalues.png" src="../../_images/normalizevalues.png" />
</div>
<ol class="arabic simple" start="7">
<li>Run the <strong>Min-Max Normalization</strong> tool by double clicking on it in the <strong>Value Transformations</strong> toolbox of the <strong>Data Preprocessing</strong> toolbox.</li>
</ol>
<div class="figure">
<img alt="../../_images/norm01.PNG" src="../../_images/norm01.PNG" />
</div>
<ol class="arabic simple" start="8">
<li>Select <strong>normVar.dbf</strong> as the <em>input database file</em>.</li>
<li>Select <strong>column</strong> as the <em>normalize by</em> field.</li>
<li>Change the <em>output database file</em> to <strong>norm01.dbf</strong>.</li>
<li>Select the columns <strong>male</strong>, <strong>female</strong>, <strong>Under_15</strong>, <strong>15_64</strong>, <strong>65_Over</strong>, <strong>Am_Indian</strong>, <strong>Asian</strong>, <strong>Black</strong>, and <strong>White</strong> in the <em>columns to normalize</em> field.</li>
<li>Click <strong>OK</strong> to run the normalization.</li>
</ol>
<p>The resulting table contains normalized values.</p>
<div class="figure">
<img alt="../../_images/norm01values.PNG" src="../../_images/norm01values.PNG" />
</div>
</div>
<div class="section" id="select-variables">
<h2>Select Variables</h2>
<p>Select the relevant variables from the database file.</p>
<ol class="arabic simple">
<li>Run the <strong>Select</strong> tool by double clicking on it in the <strong>Data Management</strong> toolbox of the <strong>Data Preprocessing</strong> toolbox.</li>
</ol>
<div class="figure">
<img alt="../../_images/select.png" src="../../_images/select.png" />
</div>
<ol class="arabic simple" start="2">
<li>Select <strong>norm01.dbf</strong> as the <em>input database file</em>.</li>
<li>Set <strong>columns</strong> as the <em>selection type</em>.</li>
<li>Change the <em>output database file</em> to <strong>demographics.dbf</strong>.</li>
<li>Select all columns except <strong>Population</strong>, <strong>Owner</strong>, <strong>Renter</strong>, and <strong>Households</strong> in the <em>columns</em> field.</li>
<li>Enable <strong>detect data types</strong>.</li>
<li>Click <strong>OK</strong> to run the selection.</li>
</ol>
<p>In table properties the value types for the columns has changed where appropriate.</p>
<div class="figure">
<img alt="../../_images/demographicfields.png" src="../../_images/demographicfields.png" />
</div>
<p>The numeric values in the table are right justified indicating that they are numbers.</p>
<div class="figure">
<img alt="../../_images/demographics.png" src="../../_images/demographics.png" />
</div>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">Detecting data types for columns requires checking the data type of each value and can be time consuming for large datasets. This  step is only necessary if performing normalizations or other calculations before using the data with a SOM.</p>
</div>
</div>
<div class="section" id="export-data">
<h2>Export Data</h2>
<p>Export the database file to the SOM data format.</p>
<ol class="arabic simple">
<li>Run the <strong>Database File to SOM_PAK Data</strong> tool by double clicking on it in the <strong>File Format Conversions</strong> toolbox of the <strong>Data Preprocessing</strong> toolbox.</li>
</ol>
<div class="figure">
<img alt="../../_images/somdat.png" src="../../_images/somdat.png" />
</div>
<ol class="arabic simple" start="2">
<li>Select <strong>demographics.dbf</strong> as the <em>input database file</em>.</li>
<li>Change the <em>output SOM data file</em> to <strong>demographics.dat</strong>.</li>
<li>Select <strong>Region</strong>, <strong>Division</strong>, <strong>State</strong>, and <strong>Year</strong> in the <em>label columns</em> field.</li>
<li>Click <strong>OK</strong> to run the export.</li>
</ol>
</div>
<div class="section" id="create-initial-som">
<h2>Create Initial SOM</h2>
<p>Creating the initial SOM.</p>
<ol class="arabic simple">
<li>Run the <strong>Create Initial SOM</strong> tool by double clicking on it in the <strong>SOM Computation</strong> toolbox.</li>
</ol>
<div class="figure">
<img alt="../../_images/mapinit.png" src="../../_images/mapinit.png" />
</div>
<ol class="arabic simple" start="2">
<li>Select <strong>demographics.dat</strong> as the <em>data for SOM</em>.</li>
<li>Select <strong>hexa</strong> as the <em>topology of map</em>.</li>
<li>Set <strong>25</strong> as the <em>x dimension</em>.</li>
<li>Set <strong>25</strong> as the <em>y dimension</em>.</li>
<li>Set <strong>init.cod</strong> as the <em>initial SOM</em>.</li>
<li>Click <strong>OK</strong> to run the creation of the initial SOM.</li>
</ol>
<p>A window will open that indicates the progress of the process.</p>
<div class="figure">
<img alt="../../_images/training.png" src="../../_images/training.png" />
</div>
</div>
<div class="section" id="train-som">
<h2>Train SOM</h2>
<p>Training the SOM.</p>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">The SOM will be trained in two steps. The first training will create the overall structure in the SOM. The second training will create the finer specialization.</p>
</div>
<ol class="arabic simple">
<li>Run the <strong>Train SOM</strong> tool by double clicking on it in the <strong>SOM Computation</strong> toolbox.</li>
</ol>
<div class="figure">
<img alt="../../_images/stage1.png" src="../../_images/stage1.png" />
</div>
<ol class="arabic simple" start="2">
<li>Select <strong>init.cod</strong> as the <em>initial som</em>.</li>
<li>Select <strong>demographics.dat</strong> as the <em>training data</em>.</li>
<li>Set <strong>4900</strong> as the <em>length of training</em>.</li>
<li>Set <strong>0.04</strong> as the <em>initial learning rate</em>.</li>
<li>Set <strong>25</strong> as the <em>initial neighborhood radius</em>.</li>
<li>Change the <em>trained SOM</em> to <strong>stage1.cod</strong>.</li>
<li>Click <strong>OK</strong> to run the training of the SOM.</li>
</ol>
<p>A window will open that indicates the progress of the process as it did with the creation of the initial SOM.</p>
<ol class="arabic simple" start="9">
<li>Run the <strong>Train SOM</strong> tool.</li>
</ol>
<div class="figure">
<img alt="../../_images/stage2.png" src="../../_images/stage2.png" />
</div>
<ol class="arabic simple" start="10">
<li>Select <strong>stage1.cod</strong> as the <em>initial som</em>.</li>
<li>Select <strong>demographics.dat</strong> as the <em>training data</em>.</li>
<li>Set <strong>49000</strong> as the <em>length of training</em>.</li>
<li>Set <strong>0.03</strong> as the <em>initial learning rate</em>.</li>
<li>Set <strong>5</strong> as the <em>initial neighborhood radius</em>.</li>
<li>Change the <em>trained SOM</em> to <strong>stage2.cod</strong>.</li>
<li>Click <strong>OK</strong> to run the training of the SOM.</li>
</ol>
</div>
<div class="section" id="calculate-u-matrix">
<h2>Calculate U-Matrix</h2>
<p>Calculate the U-matrix of a SOM.</p>
<ol class="arabic simple">
<li>Run the <strong>Calculate U-matrix</strong> tool by double clicking on it in the <strong>SOM Computation</strong> toolbox.</li>
</ol>
<div class="figure">
<img alt="../../_images/umatrix.PNG" src="../../_images/umatrix.PNG" />
</div>
<ol class="arabic simple" start="2">
<li>Select <strong>stage2.cod</strong> as the <em>input SOM</em>.</li>
<li>Change the <em>output U-matrix database file</em> to <strong>Umatrix.dbf</strong>.</li>
<li>Click <strong>OK</strong> to calculate the U-matrix</li>
</ol>
<div class="figure">
<img alt="../../_images/umatrixvalues.PNG" src="../../_images/umatrixvalues.PNG" />
</div>
</div>
<div class="section" id="project-data-onto-som">
<h2>Project Data onto SOM</h2>
<p>Project the data onto the SOM.</p>
<ol class="arabic simple">
<li>Run the <strong>Project Data onto SOM</strong> tool by double clicking on it in the <strong>SOM Computation</strong> toolbox.</li>
</ol>
<div class="figure">
<img alt="../../_images/bmu.png" src="../../_images/bmu.png" />
</div>
<ol class="arabic simple" start="2">
<li>Select <strong>stage2.cod</strong> as the <em>SOM</em>.</li>
<li>Select <strong>demographics.dat</strong> as the <em>data to project</em>.</li>
<li>Change the <em>projected data</em> to <strong>demographics.bmu</strong>.</li>
<li>Click <strong>OK</strong> to project the data onto the SOM.</li>
</ol>
<p>A window will open that indicates the progress of the process as it did with the creation of the initial SOM.</p>
</div>
<div class="section" id="create-som-shapefile">
<h2>Create SOM Shapefile</h2>
<p>Creating the SOM shapefile.</p>
<ol class="arabic simple">
<li>Run the <strong>SOM to Shapefile</strong> tool by double clicking on it in the <strong>SOM Visualization</strong> toolbox.</li>
</ol>
<div class="figure">
<img alt="../../_images/somshape.png" src="../../_images/somshape.png" />
</div>
<ol class="arabic simple" start="2">
<li>Select <strong>stage2.cod</strong> as the <em>SOM</em>.</li>
<li>Select <strong>polygon</strong> as the <em>shape type</em>.</li>
<li>Change the <em>SOM shapefile</em> to <strong>stage2.shp</strong>.</li>
<li>Set <strong>demographics.dat</strong> as the <em>SOM data for variable names</em>.</li>
<li>Enable <em>label SOM with data labels</em></li>
<li>Set <strong>Umatrix.dbf</strong> as the <em>U-matrix</em>.</li>
<li>Click <strong>OK</strong> to create the SOM shapefile.</li>
</ol>
</div>
<div class="section" id="create-data-shapefile">
<h2>Create Data Shapefile</h2>
<p>Creating the data shapefile.</p>
<ol class="arabic simple">
<li>Run the <strong>Projected Data to Shapefile</strong> tool by double clicking on it in the <strong>SOM Visualization</strong> toolbox.</li>
</ol>
<div class="figure">
<img alt="../../_images/bmushape.png" src="../../_images/bmushape.png" />
</div>
<ol class="arabic simple" start="2">
<li>Select <strong>demographics.bmu</strong> as the <em>projected data</em>.</li>
<li>Select <strong>point</strong> as the <em>shape type</em>.</li>
<li>Change the <em>projected data shapefile</em> to <strong>bmu.shp</strong>.</li>
<li>Select <strong>demographics.dat</strong> as the <em>label from SOM data</em>.</li>
<li>Select <strong>random around center</strong> as the <em>placement</em>.</li>
<li>Click <strong>OK</strong> to create the data shapefile.</li>
</ol>
</div>
<div class="section" id="group-data-shapefile">
<h2>Group Data Shapefile</h2>
<p>Grouping the shapes in the data shapefile.</p>
<ol class="arabic simple">
<li>Run the <strong>Group Shapes</strong> tool by double clicking on it in the <strong>SOM Visualization</strong> toolbox.</li>
</ol>
<div class="figure">
<img alt="../../_images/trajectory.png" src="../../_images/trajectory.png" />
</div>
<ol class="arabic simple" start="2">
<li>Select <strong>bmu.shp</strong> as the <em>input shapefile</em>.</li>
<li>Select <strong>State</strong> as the <em>group by column</em></li>
<li>Select <strong>polyline</strong> as the <em>group type</em>.</li>
<li>Select <strong>maximum</strong> as the <em>value type</em>.</li>
<li>Change the <em>output shapefile</em> to <strong>trajectories.shp</strong>.</li>
<li>Select <strong>Year</strong> as the <em>sort by column</em>.</li>
<li>Click <strong>OK</strong> to create the trajectories.</li>
</ol>
</div>
<div class="section" id="create-extent-shapefile">
<h2>Create Extent Shapefile</h2>
<p>Creating the extent shapefile.</p>
<ol class="arabic simple">
<li>Run the <strong>Create Extent Shapefile</strong> tool by double clicking on it in the <strong>Utilities</strong> toolbox.</li>
</ol>
<div class="figure">
<img alt="../../_images/extent.PNG" src="../../_images/extent.PNG" />
</div>
<ol class="arabic simple" start="2">
<li>Select <strong>stage2.shp</strong> as the <em>input shapefile</em>.</li>
<li>Change the <em>output shapefile</em> to <strong>extent.shp</strong>.</li>
<li>Click <strong>OK</strong> to create the extent shapefile.</li>
</ol>
</div>
<div class="section" id="visualization">
<h2>Visualization</h2>
<p>Visualizing the SOM and projected data.</p>
<ol class="arabic simple">
<li>Open <strong>tutorial.mxd</strong>.</li>
</ol>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">Your map will not be identical, but should be very similar. The frames may appear rotated due to the initial random numbers used.</p>
</div>
<div class="figure">
<img alt="../../_images/tutorial.png" src="../../_images/tutorial.png" />
</div>
<p>The large map shows the trajectory of each state across the SOM over time with a base of the U-matrix, a measure of distortion. The trajectories are color coded by census division, which are shown in the lower right. The other frames contain the component planes, each showing the neuron weights for one variable across the entire SOM.</p>
<p>When examining the demographic trajectories of each state note that each shift in the trajectory corresponds to a census year and that at the end of the trajectory is an arrow that represents the year 1990. Parallel trajectories indicate a similar change in demographics over time. Parallel trajectories are particularly evident within the South Division (West South Central Region, East South Central Region, and South Atlantic Region) and Northeast Division (Middle Atlantic Region and New England Region). This demonstrates spatial autocorrelation and is consistent with the demographic changes over the last century. In the Northeast Division, the parallel trajectories split 40 years ago mainly into coastal and land locked areas with New York and New Jersey similar to each other, but dissimilar to the other coastal states.</p>
<p>When examining component planes you are seeing how the SOM allocates location based on that variable. In this map, darker color means high values and lighter color means low values. You can see that the female component plane is very dark in one corner and light in the opposite corner with a gradual change between the two. Conversely the male component plane is very dark in the opposite corner and has a similar pattern of gradual change. When comparing component planes to each other you can see how the SOM weights the variables in the same location and thus derive a relationship between them. You can see that that female and male have an inversely proportional relationship in the SOM that corresponds with reality, that is that a high number of females inherently means a low number of males and vice versa.</p>
</div>
</div>


          </div>
      </div>
      <div class="clearer"></div>
    </div>
    <div class="related">
      <h3>Navigation</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="../../genindex.html" title="General Index"
             >index</a></li>
        <li class="right" >
          <a href="../../py-modindex.html" title="Python Module Index"
             >modules</a> |</li>
        <li class="right" >
          <a href="uiEmailMe.html" title="Send Email"
             >previous</a> |</li>
        <li><a href="../../index.html">SOM Analyst v1 documentation</a> &raquo;</li> 
      </ul>
    </div>
    <div class="footer">
        &copy; Copyright 2010, Martin Lacayo.
      Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> 1.0.7.
    </div>
  </body>
</html>